A novel key partitioning schema for efficient execution of MapReduce applications

MapReduce and its open source implementation, Hadoop, are the prevailing platforms for big data processing. MapReduce is a simple programming model for performing large computational problems in large-scale distributed systems. This model consists of two major phases: Map and Reduce. Between these two main phases, partitioner part is embedded which distributes produced keys by Map tasks among Reduce tasks. When the amount of keys and their associated values, which are called intermediate data, is huge, this part has significant impact on execution time of Reduce tasks, and consequently, completion time of jobs. In this paper, we present a network and resource aware key partitioner to decrease the execution time of MapReduce jobs. Using sampling, our algorithm finds the distribution of keys in intermediate data. Then, considering aforementioned distribution, the amount of each key on each machine, the placement of Reduce tasks on machines and the network bandwidth between machines, our algorithm assigns keys to Reduce tasks to decrease the total execution time of job. Our experiments show that our approach can improve completion time of Reduce phase and job execution time by up to 52% and 31% respectively compared with Hadoop default partitioner and can find the solution within 8% of ideal partitioner.